Quack Remedies, False Prophets and Unwarranted Claims

Course Code: PSDI 2190                                                                                   Fall 2007
Professor: Fedorchak                                                                     Plymouth State University

This course identifies common myths and misunderstandings about the scientific process and its rules of evidence, and helps students separate real science from pseudoscience, proven cures from quack remedies and legitimate dangers from media scares.

Required Text:  "How to Think Straight About Psychology" by Keith Stanovich (8th Edition; available at the College Bookstore and possibly also the Plymouth Book Exchange.  [Note:  You could get away with using the 7th edition, but you might have to find the page numbers I assign on your own]).   This book was originally written for a behavioral science audience, but is now used in a variety of fields.   The focus of the book is about separating science from pseudoscience - wherever it is found.  To accomplish this goal, Stanovich uses examples from the areas of Psychology,  Epidemiology, Physics, Statistics/Probability and Medicine, as well as from other disciplines.  The text readings will be supplemented with primary research articles, websites, newspaper clippings and videotapes, etc., that will be handed out, referenced or viewed throughout the semester.

Office Hours (Fall 2007):  Monday and Wednesday: 12:15 to 1:15PM; Tuesday: 2:30 to 3:30 PM; Thursday: 1 to 2 PM; Other times by appointment.

Course objectives/goals: 

1) You should develop a clear understanding of the concept of scientific evidence, and learn how this standard can be applied to the variety of claims we are confronted with on a daily basis.

2) You will gain an appreciation for the variety of invalid   arguments often used to "contradict" scientific claims about causation and correlation, and you will learn why they are invalid.

3) You will develop an understanding about why reasonable people put forth such arguments, and why so many others believe them (i.e., why the arguments are so persuasive).

4) You will learn about the limits of what science can tell us about causation and prediction, especially in fields like psychology and biomedicine where the data tend to be highly variable.  From this you should gain an appreciation for why research findings so often seem contradictory, even when they are not.

5)  You will develop an appreciation for the ubiquitous role that probability  plays in evaluating claims about causation and correlation, and will learn why many claims can only be stated in probabilistic terms (e.g., we can say with a certain probability   that a certain proportion  of people exposed to this toxin will get cancer; yet we usually can't say the toxin caused  it, or even who  will get it.).

6) You will gain an appreciation for the impossibility of eliminating bias, and the importance of using data collection techniques that neutralize it.

7)  You will come to appreciate how readily people will state or even publish almost any claim with minimal evidence to  back it up, and how easily the media will often pick up and repeat such claims.

Grading:  Grades will be based on three (3) exams (each worth 22% of grade),  one (1) Article Analysis/Reaction (each worth 17% of grade), and one (1) team data collection exercise and individual report (17% of grade).  

The Exams:  The 3 exams will be spaced at approximately equal intervals, with the last exam scheduled for the final exam time and day (the data collection presentations, as noted below, will occur on the last few days of class).

The Article Analysis/Reaction:  During the semester, each student will locate an article that either promotes or is skeptical of some type of claim and write a 3+ page (double-spaced and typed) analysis of and personal reaction to it.  These short papers should allow you to become familiar with specific areas of controversy and develop an understanding of the types of problems associated with evaluating their particular claims.  Each person will submit an ‘intended topic’ e-mail, as I’d like everyone to write about a slightly different topic (or if at least use a different article on the same topic).   First person to ‘claim’ a particular topic/article will get to use it.

Data Collection/Class Presentation:

1. Do they really  give you fewer fries when you go through the drive-through?
2. Does my dog or cat really  know when I'm coming home?
3. Does it really  matter if you use one egg or two, or water instead of milk, in the recipe?
4. Do people really  tip more if they see "Have a nice day!" written on their check?
5. Does the car heater really  warm up faster if you press the air re-circulation button?
6. Does coffee really  taste better if you start with ice cold  water?

People often say they know something is true because of their own personal experience.   In the past, I've heard or said all of the things listed above and more.   Unfortunately, most such claims are based on one or two memorable experiences that may or may not be typical; or worse, may not have happened as often or at all.   The goal of this exercise is to use some basic principles of data collection (e.g., blind measuring, random sampling) to cut through the natural variability  of such common events and rise above the biases and expectations that can distort our detection, perception and memory of them.  In the end, we'll have a clearer understanding of whether such claims are true or false, an eminently defensible basis for how we "know" this, and - most importantly - a better appreciation for the general applicability of the scientific method. 

Teams of students (4 per/team) will identify an everyday question or assumption that could be answered or verified through a carefully planned set of observations.   You will plan the data collection, identify potential problems, then carry out the observations.  This project will be conducted throughout the entire second half of the semester.  On one of the last days of class you will present your findings (as a team) then submit individual write-ups describing your data collection process and results.  You will also be asked to anonymously rate your own and each of your teammates’  contributions to the project.

General Outline of Topics

-  Course Overview, Dissection of course title.
-  Open-ended discussion of common claims, and some reasons people do or do not believe them.

False Prophets  (Prediction)

-  Systems of Prediction can seem valid, when they are not; or can appear invalid, when they are actually legitimate.  Why?  Answer:  Variability.

1. Confirmation Biases and Illusory Correlations: why we “see” relationships that do not exist.   [Demonstrations: Dowsing for Dots;  2 X 2 matrix; Punxsutawney Phil – for homework]
2. VIDEO CLIP: Dowsing

-  The undeserved persuasive power of Anecdotes and Testimonials: When friends' testimonials contradict hard data.
-  VIDEO CLIP:  Iridology, Live Cell Analysis & Applied Kinesiology - a Dateline Special.   [Exercise (via powerpoint):  Iridology vs. Coinology; p < .05]
- Predicting better than chance;  estimating chance levels of performance for simple and complex systems of prediction.  
-  VIDEO:  Secrets of the Psychics - a NOVA presentation featuring the Amazing Randi.
-  Determination of teams and questions.

Reading:  Chapters 1 through 4.

EXAM #1

Quack Remedies (Causation/Control)

-  From Prediction to Causation: The case of Beta-Carotene.
-  True Experiments, Placebo Effects and other Expectancy  confounds.
-  VIDEO:  Prisoners of Silence (Facilitated Communication:  a modern-day Clever Hans.)
-  Double-Blind methods: the gold standard  of medical (and other) research.
-  VIDEO CLIP:  Therapeutic Touch.
-  Two common myths: the victim as expert ("Lets ask Uncle Ralph what caused his cancer!") and the MD as expert. 
-   Epidemiological studies: what they can (usually) tell us. 
-   The Big Picture.

Reading:  Chapters 5 through 9.

EXAM #2

Unwarranted Claims (possible topics)

-  Surveys and Self-Selection problems: The 7/11 Poll vs. Scientific Polls.
-  The Base Rate Issue:  Vasectomy causes prostate cancer ?
-    VIDEO:  Junk Science (ABC special w/John Stossel)

1. The Fluoride Study (Kingston, NY  vs. Newburg, NY)
2. Cancer near nuclear power plants (w/ Sir Richard Doll)

-    Fishing Expeditions (w/ fisherman Jay Gould, author of “Deadly Deceit”).
-    Discussion of two movies: Lorenzo’s Oil and A Civil Action.   (And now for… the rest of the story.)
-    Psychics (e.g., Sylvia Brown) and Mediums (e.g., John Edward) – how they can “seem” to know all about you and/or your deceased relatives. 

1. Alternative Medicine: NIH's NCCAM; DSHEA (1994); FTC vs. FDA
2. Bean Pollsters:  a classroom simulation/election polling special.

Reading: Chapters 10 and 11; plus  Fedorchak's Essays on Risk and Probability (to be posted on WebCT)

Data Collection Presentations (Final 2 regular classes)  

Final note:  Throughout this course as we address the various types of claims, we will continually move back and forth between the logic of scientific evaluation and the use of probability.   As the course progresses, students should move away from rote memorization of the specifics of testing each different claim and, instead, begin to recognize the similarities in how each question is approached.  Ideally, each student will come away from the course with a general set of principles that can be used to evaluate any testable claim, and the confidence to use them.